Using Item-type Performance Covariance to Improve the Skill Model of an Existing Tutor

Using data from an existing pre-algebra computer-based tutor, we analyzed the covariance of item-types with the goal of describing a more effective way to assign skill labels to item-types. Analyzing covariance is important because it allows us to place the skills in a related network in which we can identify the role each skill plays in learning the overall domain. This placement allows more effective and automatic assignment of skills to itemtypes. To analyze covariance we used POKS (partial order knowledge structures) to analyze item-type outcome relationships and Pearson correlation to capture item-type duration relationships. Hierarchical agglomerative clustering of these item-types was also performed using both outcome and duration covariance patterns. These analyses allowed us to propose improved skill labeling that removes irrelevant item-types, clusters related types, and clarifies the optimal temporal ordering of these clusters during practice. 1 Carnegie Learning’s Bridge to Algebra Cognitive Tutor Our goal was to examine a large dataset (>9 million problems step performances) to determine a skill model that we could subsequently use to produce improvements to a Cognitive Tutor [1]. While the tutor had a skill model coded by human domain experts, we felt that it would be useful to develop alternative methods of coding skills that might be less vulnerable to the possibility of human error and bias. The dataset was provided by Carnegie Learning Inc. from the Bridge to Algebra Cognitive Tutor for the 2006-2007 school year [2]. This tutor works by providing a systematic coverage with 44 units of prealgebra content each of which has several sections. These sections consist of a problem type, which is composed of several steps or “item-types” which repeat over a sequence of similar problems. Data from this tutor included times of problem step actions and outcomes of step actions. Because, the human skill model was coded at this step level, we also choose to examine student performances at the step level (henceforth these individual steps in problems in sections in units will be called item-types). By using this level of analysis we will be able to qualitatively compare the skill model that the tutor uses with the skill model suggested in our analyses. 2 Areas of improvement Human coding and selection of skills in a tutor may introduce the following three possible problems which our datamining algorithm addresses. As we discuss skills, it should be assumed that we are considering skills more generally as latent variables that may represent knowledge components important to a concept, rule application, or other learnable proficiency necessary for responding to an item-type. 2.1 Problem of irrelevant skills This issue refers to labeling an item-type as needing a skill despite it having a weak relation with other proficiencies that the tutor is focused upon. These weak relationships may be due to the fact that the action is either too simple (probability of success near to 1), too hard (probability of success near 0) or that the task is irrelevant to (independent of) the more complex related target proficiencies. In the case of simple tasks, the skill should not be labeled because the item-type should not be included in the tutor since performance is already at/near the desired level and time is wasted in further repetition. In the case of the difficult task we can suppose that some actions should not be included in a tutor because they result in poor learning and frustrate students. In both of these cases, our method will ignore these skills since a skill lacking variance cannot covary. In the case of an independent task it clearly makes little sense to include the item-type in the tutor unless the data suggests it is associated with a target skill of the domain. 2.2 Problem of skill redundancy This is the issue of labeling two different item-types with two independent skill labels despite strong similarities in the skill involved. When a human decides to label a skill there is always the question whether the action is statistically equivalent with another action in the tutoring system being analyzed. Of course, this is a very difficult problem to judge since it depends on whether variation in the actions is different enough to produce difference in performances that are of practical significance. Because if this human experts may tend to label skills as different despite strong similarities. This leads to a model that tends to have poor intuitions about the how much practice to give a particular skill because when similar item-types are treated as independent the model can make no conclusions about how much practice to give the second one depending on performance with the first one. Thus, a tutor with redundant skills will be forced to give too little or too much practice because it is ignorant of the underlying skill overlap. 2.3 Problem of skill ordering It is generally thought that a curriculum has a fixed order in which earlier skills form part of later whole skills or target performances, but this order may be difficult to identify. The importance of curriculum order may be due to a general benefit for training part skills before introducing whole skills. Wightman and Lintern [3] have described two ways of splitting a whole task into components tasks: segmentation and fractionation. Segmentation splits the whole task into sequential steps, and fractionation splits the whole task into time-shared parts. Optimal curriculum ordering may require some parttraining in either case. With segmented skills, part-training may allow more efficient targeted training that can avoid already learned parts of the whole task. In the case of fractionated skills, part-training may provide the initial skill that enables the learner to successfully practice the whole skill. Because our method can identify prerequisites, we can hope that it will help us answer questions about the ordering of skills in a principled way.